Capture MCP tool calls in PostHog by masnwilliams · Pull Request #128 · kernel/kernel-mcp-server

masnwilliams · 2026-07-29T19:07:08Z

Summary

Instruments the MCP server with PostHog MCP analytics (@posthog/mcp, pinned to 0.10.1) so every inbound tools/call, tools/list, and initialize is captured as a $mcp_* event — tool name, latency, error state, client name/version, session.

What that buys us: per-tool call volume, error rate and latency percentiles, which clients connect, and which advertised tools never get called. Today we ship tool changes blind — kernel API traffic tagged X-Source: mcp-server shows endpoints, not tools.

src/lib/mcp/analytics.ts — module-scope posthog-node client plus instrument() wiring. identify attributes events to the Clerk user id when the request carries a JWT; API-key requests stay session-scoped rather than getting a made-up identity.
src/app/[transport]/route.ts — instrument inside the createMcpHandler callback, mint a session id on the handshake (below), and drain the queue with after() so the flush runs once the response is already on the wire. Capture adds no latency to a tool call.
.env.example — POSTHOG_PROJECT_TOKEN and POSTHOG_HOST. With no token set the whole thing is a no-op, so deploys and local dev without PostHog credentials are unaffected (dev logs a one-time error so the miss isn't silent).

POSTHOG_PROJECT_TOKEN and POSTHOG_HOST are set on the Vercel project (production → prod PostHog project, preview → staging).

Sessions

mcp-handler runs the streamable-HTTP transport statelessly and answers over SSE, so it never issues an Mcp-Session-Id. Left alone that means one PostHog session per HTTP request, and $mcp_client_name / $mcp_client_version — only sent at initialize — missing from every later event.

mintMcpSessionId handles it the way the SDK documents for SSE: on an initialize POST it mints a session token carrying a fresh session id plus the client name and version, injects it on the inbound request (so the handshake event lands in that session too) and echoes it back on the response. Clients replay the header, and any instance decodes the same values out of it — no session store, no sticky routing. The stateless transport ignores an incoming session id (validateSession short-circuits when sessionIdGenerator is undefined), so this can't produce 400/404s. Mcp-Session-Id is added to the CORS allow/expose lists so browser-based clients can round-trip it.

No payloads leave the server

beforeSend drops $mcp_parameters and $mcp_response on every event.

Neither side of a call is safe to capture here. Results are serialized through jsonResponse, so a response reaches the SDK as one JSON string and its key-name redaction can't see inside it — a newly created API key, a TOTP code, or a CDP URL would go out verbatim. Arguments are free-form on many tools: credential field maps (manage_credentials values, manage_auth_connections fields), browser_curl headers and body, computer_action typed text, exec_command commands, execute_playwright_code source. An allow-list of "credential tools" would always be one tool behind, so the payloads go entirely and what remains is call metadata.

If we later want an argument breakdown (e.g. which action is most used), the safe shape is an explicit allow-list of individual parameter keys.

Deliberately left off

instrument() can inject a required context argument into every tool schema to capture what the agent was trying to do ($mcp_intent). It's the most interesting signal, but it changes a public tool surface — a required field plus a 15-25 word instruction on all 17 toolsets — so this ships with context: false. Worth turning on as its own change once the basic numbers are landing.

The SDK is pre-1.0 (0.x), so event names and properties can change in minor releases — hence the exact-version pin.

Verification

bunx tsc --noEmit passes (what CI runs).
Ran the real route locally against a non-production PostHog project. A handshake plus a tools/list and a tools/call replaying the minted header land as three events under one session id — the exact id inside the token — each carrying client name and version. A control request that doesn't replay the header lands in its own session with a null client, which is what the currently deployed staging server produces for every event.
Payload dropping verified the same way: events for a call carrying a canary argument value have neither $mcp_parameters nor $mcp_response present, and no canary string anywhere in the event.
bun run build fails locally on this branch and on main alike (page data collection for /authorize without Clerk/Redis env), so that failure is unrelated.

Note

$mcp_server_name currently reads mcp-typescript server on vercel — mcp-handler's default serverInfo, which is also what agents see. Passing serverInfo: { name: "kernel-mcp-server", version } to createMcpHandler would fix both, but it changes the advertised server identity so it's left out of this PR.

Note

Medium Risk
Changes run on every authenticated MCP GET/POST and send telemetry externally, but capture is gated on env, strips sensitive payloads, and does not alter auth or tool schemas.

Overview
Adds PostHog MCP analytics so initialize, tools/list, and tools/call emit $mcp_* events (tool name, latency, errors, client info) when POSTHOG_PROJECT_TOKEN is set; without it, instrumentation is a no-op.

New src/lib/mcp/analytics.ts wires @posthog/mcp with metadata-only capture: a strict property allow-list strips tool arguments, responses, and error text; intent injection and exception autocapture stay off; person identification is disabled (identify: null).

The MCP transport route instruments the handler, schedules after(flushMcpAnalytics) so flushes don’t block responses, and on streamable HTTP initialize mints an Mcp-Session-Id token (injected on the request and echoed on the response) so handshake and follow-up calls share one analytics session. CORS preflight/response headers allow clients to round-trip that header.

.env.example documents POSTHOG_PROJECT_TOKEN and POSTHOG_HOST; dependencies add pinned @posthog/mcp and posthog-node.

^{Reviewed by Cursor Bugbot for commit 96adad9. Bugbot is set up for automated code reviews on this repo. Configure here.}

vercel · 2026-07-29T19:07:10Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
mcp	Ready	Preview	Jul 30, 2026 4:27pm
mcp (staging)	Ready	Preview	Jul 30, 2026 4:27pm

socket-security · 2026-07-29T19:07:25Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	posthog-node@5.46.1
	@posthog/mcp@0.10.1

View full report

…ment

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d9b21ea. Configure here.}

dcruzeneil2

Thanks for putting this together. The overall direction is strong, and there are several thoughtful decisions here, particularly disabling context capture, pinning the pre-1.0 SDK version, and removing tool arguments and responses across all tools. The session propagation approach is also a sensible solution for a stateless deployment.

I reviewed the implementation and traced the relevant behavior through the pinned versions of @posthog/mcp and posthog-node. I found a few privacy issues that I think should be addressed before approval, followed by some non-blocking correctness and analytics considerations.

Blockers

Application-wide exception capture is enabled

enableExceptionAutocapture: true is configured on the posthog-node client. In this location, it installs process-level handlers for uncaught exceptions and unhandled promise rejections.

This is not limited to MCP tool calls. It can capture failures from unrelated code running in the same server process. These events also bypass the MCP-specific beforeSend filtering, so their messages and stack traces are not protected by the payload-removal logic in this PR.

This option should be removed from the PostHog client configuration.

Failed tool responses can still leak through error properties

Removing $mcp_parameters and $mcp_response is a good safeguard. However, @posthog/mcp processes failed tool results separately.

The SDK can extract text from a failed result and send it as $mcp_error_message. It can also create a separate $exception event containing the same message. Neither is removed by the current filter.

This means free-form error text can still leave the server even though the original response is removed. Error messages may contain upstream response details or user-provided values.

To preserve a metadata-only boundary, free-form error messages and companion exception events should not be captured. A safe categorical field such as the error type can still be retained.

The payload filter permits unknown properties by default

The current filter removes two known properties and sends everything else:

delete properties[PostHogMCPAnalyticsProperty.Parameters];
delete properties[PostHogMCPAnalyticsProperty.Response];

This works with the currently pinned SDK schema. However, if a future SDK version renames one of these properties or introduces another payload-bearing property, that data would be sent automatically.

Because this integration is intended to send metadata only, it should use an explicit allow-list of approved metadata properties. New or unknown properties should be discarded by default.

Non-blockers

Analytics identity and grouping

The selected `distinct_id` may fragment user profiles

The integration uses the Clerk subject as the PostHog distinct_id.

That is only correct if the Clerk subject is the canonical identity used by every other producer writing to the same PostHog project. If another producer identifies the same person using a different stable ID, PostHog will create separate profiles for the same person.

This would make cross-product user journeys and user-level adoption analysis unreliable. The selected distinct_id should be checked against the identity convention of the target PostHog project before merging.

API-key traffic lacks a stable customer identity

The identity callback returns no identity for API-key requests. Those events may therefore be associated only with an analytics session rather than a stable customer identity.

Tool-level reporting will still work, but usage across multiple sessions cannot be reliably attributed to the same customer.

Organization grouping is not included

The integration does not attach an organization group. This limits the ability to report on MCP adoption and usage at the customer or organization level.

The intended identity and grouping behavior should be documented, even if implementation is left as follow-up work.

Session and transport correctness

Legacy SSE sessions may be split

The custom session ID is minted for any POST containing an initialize request. This includes initialization requests sent through the legacy /message transport.

Legacy SSE clients do not necessarily replay the new session header. The initialize event may therefore use the newly minted ID while later tool events use the existing SSE transport session ID.

This would place initialization in one analytics session and subsequent tool calls in another. Custom session minting should be limited to the streamable HTTP transport.

Protocol version is not preserved

The session token stores the session ID, client name, and client version, but not the protocol version from the initialize request.

Later stateless requests rely on this token for session information, so their analytics events cannot consistently include the MCP protocol version.

Request reconstruction has no fallback

The initialize request is reconstructed to inject the session header. The normal path should work, but reconstruction does not preserve every part of the original request, such as its abort signal.

There is also no fallback if reconstruction fails. Since initialization is required before a client can use the server, an analytics-related failure on this path could prevent the MCP session from starting.

Performance

Large payloads are processed before they are removed

The SDK sanitizes, truncates, and measures tool arguments and responses before beforeSend removes them.

Large tool results may therefore consume CPU and memory even though they are never transmitted. Avoiding this work likely requires an upstream SDK option that disables payload capture earlier in the pipeline.

Recommendation

This is a useful addition, and the underlying approach looks solid. I would request changes for the three privacy blockers:

Remove application-wide exception autocapture.
Prevent free-form tool error messages and companion exception events from being sent.
Replace the property deny-list with an explicit metadata allow-list.

I would also confirm the distinct_id strategy before merging because an identity mismatch would make user-level analytics misleading and difficult to repair later.

Once the privacy boundaries are tightened, I would be comfortable approving. The remaining session, transport, grouping, and performance findings can reasonably be handled as follow-up work.

Removes process-level exception autocapture, drops the $exception sibling and free-form error text a failed tool call produced, and replaces the payload deny-list with an allow-list of the properties this integration intends to send. Also stops identifying on the Clerk subject, which is not the identity other producers in these PostHog projects use, carries the negotiated protocol version in the session token, limits session minting to the streamable HTTP transport, and forwards the abort signal when reconstructing the initialize request.

masnwilliams · 2026-07-30T16:27:27Z

Thanks — this was a careful read, and all three blockers were real. Traced each one through the pinned @posthog/mcp@0.10.1 / posthog-node@5.46.1 before changing anything; fixes are in 96adad9.

Blockers

1. Application-wide exception autocapture — removed. Confirmed in posthog-node: enableExceptionAutocapture reaches ErrorTracking.startAutocaptureIfEnabled(), which registers process.on('uncaughtException') and process.on('unhandledRejection'). Worse than described: makeUncaughtExceptionHandler counts other uncaughtException listeners and, when it's the only one, calls onFatalError → client.shutdown(2000) → process.exit(1). An analytics dependency with a say in process termination. Option dropped; posthog-node defaults it to false.

2. Free-form error text — no longer sent, and this had already happened. Two channels, both confirmed in the SDK: $mcp_error_message comes from $exception_list[0].value on the primary event (posthog-events.ts), and buildExceptionEvent fans out a $exception sibling that spreads the same $exception_list. Both are built from captureException(result), which for an isError CallToolResult concatenates the tool's text content — so it's the tool's error text verbatim, not a type name.

Not theoretical: the staging project already holds 7 $mcp_tool_call events with $mcp_error_message and 7 matching $exception events with $lib = posthog-node-mcp. Benign text in this case, but the mechanism is exactly as you describe. Now enableExceptionAutocapture: false on instrument() (no sibling built), beforeSend drops any $exception payload that reaches it, and $mcp_error_message isn't on the allow-list. $mcp_error_type is kept as the categorical field.

3. Deny-list → allow-list — done. beforeSend now keeps only the properties this integration intends to send and deletes everything else, so a renamed or newly added payload property can't start flowing on an upgrade. Kept: source, session id, tool name/description/category, listed tool names, resource name, duration, server name/version, client name/version, protocol version, $mcp_is_error, $mcp_error_type, plus $process_person_profile and $groups. Deliberately excluded: $mcp_parameters, $mcp_response, $mcp_error_message, and $mcp_intent / $mcp_intent_source — which also means turning intent capture on later has to be an explicit change here, not a silent side effect.

Verified end to end against the staging project: a 4-event session (initialize, tools/list, one successful and one failed tool call) carries only the allow-listed keys plus the ones posthog-node stamps server-side ($lib, $ip, $sent_at, …). No $mcp_error_message, no $exception event, and a canary string passed as a tool argument appears nowhere.

Non-blockers

Identity — you were right to flag it, and the answer is that the Clerk subject is wrong. Checked against the target project rather than assuming: over 7 days, 0 events use a Clerk-style user_… distinct id and 32.8M use a 24-character Kernel id, with person properties (name, email) attached from the dashboard and the Go API. Identifying on payload.sub would have created a second profile for every human already in there, which is painful to unwind after the fact.

The MCP server can't resolve the Kernel user id today (nothing in @onkernel/sdk exposes it), so identify is now null: events are session-scoped, $process_person_profile = false, no persons created. Non-destructive, and adding real identity later is purely additive. That also folds in your API-key point — no traffic gets a synthetic identity, so JWT and API-key traffic are treated the same rather than one being half-attributed.

Organization grouping — follow-up, and more standard than "nice to have": 31.5M of 33.2M production events in the last 7 days carry $groups. $groups is on the allow-list so wiring it up is a one-line change once the org id is resolvable by the same means as the user id. Both are documented in the code comment on identify.

Legacy SSE session splitting — fixed. Minting is now gated to the streamable HTTP transport (/mcp); an initialize arriving on the legacy /message path is left alone.

Protocol version — fixed. encodeSessionId accepts protocolVersion, so it's now carried in the token. Confirmed: $mcp_protocol_version is present on tools/list and tools/call events, where it was previously null after the handshake.

Request reconstruction — abort signal now forwarded via signal: req.signal. I didn't add a fallback: reconstruction is two constructor calls that run only after the body has already parsed as JSON, and a failure there would break the fallback path the same way. If you'd still prefer a guard, say so and I'll add one.

Payload processing cost — upstream, and filed. Confirmed the ordering you describe: processMcpEvent runs sanitizeEvent and truncateEvent over the full payloads before beforeSend sees them, so the work happens regardless. It needs an SDK option; I've sent PostHog a request for a capturePayloads: false-style flag that skips payload capture before sanitization. Until it exists this is wasted CPU on large results, not a correctness or privacy issue.

dcruzeneil2

Thanks for the detailed response and for addressing the findings so thoroughly. I re-reviewed the latest commit and confirmed that all three blockers are resolved:

Process-wide exception autocapture has been removed.
Free-form tool error messages and companion $exception events are no longer sent.
Event properties now use an explicit metadata allow-list.

The identity change is also a safe choice for now. Keeping events session-scoped avoids creating incorrect person profiles while leaving proper user and organization attribution as additive follow-up work.

The legacy SSE session handling, protocol version propagation, and abort signal forwarding have also been addressed. I agree that payload preprocessing belongs upstream and that request reconstruction fallback does not need to block this PR.

Everything needed for this PR is addressed. Approving.

Capture MCP tool calls in PostHog

5fb3271

vercel Bot deployed to Preview July 29, 2026 19:07 View deployment

masnwilliams marked this pull request as ready for review July 29, 2026 19:11

Flush analytics after the response and drop the injected context argu…

981d564

…ment

masnwilliams marked this pull request as draft July 29, 2026 19:16

vercel Bot deployed to Preview July 29, 2026 19:17 View deployment

cursor Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread src/lib/mcp/analytics.ts

masnwilliams marked this pull request as ready for review July 29, 2026 19:27

Drop response payloads and credential arguments before send

b1a9ccf

masnwilliams marked this pull request as draft July 29, 2026 19:35

vercel Bot deployed to Preview July 29, 2026 19:35 View deployment

masnwilliams marked this pull request as ready for review July 29, 2026 19:36

vercel Bot deployed to staging July 29, 2026 19:36 View deployment

cursor Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread src/lib/mcp/analytics.ts

Drop auth connection submit parameters before send

cf6d634

vercel Bot deployed to staging July 29, 2026 21:55 View deployment

vercel Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread src/lib/mcp/analytics.ts Outdated

cursor Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread src/lib/mcp/analytics.ts

Mint a session id so stateless requests share one MCP session

47a73cb

vercel Bot deployed to staging July 29, 2026 22:12 View deployment

Stop capturing tool arguments alongside responses

01a6476

vercel Bot deployed to staging July 29, 2026 22:17 View deployment

cursor Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread src/app/[transport]/route.ts

Expose the session header on the response that sets it

d9b21ea

vercel Bot deployed to staging July 29, 2026 22:35 View deployment

cursor Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread next-env.d.ts Outdated

Restore generated next-env types path

1de2e9a

vercel Bot deployed to staging July 29, 2026 22:47 View deployment

masnwilliams requested a review from dcruzeneil2 July 29, 2026 23:00

dcruzeneil2 requested changes Jul 30, 2026

View reviewed changes

vercel Bot deployed to staging July 30, 2026 16:27 View deployment

masnwilliams requested a review from dcruzeneil2 July 30, 2026 17:25

dcruzeneil2 approved these changes Jul 30, 2026

View reviewed changes

masnwilliams merged commit 72d469a into main Jul 30, 2026
10 checks passed

masnwilliams deleted the hypeship/posthog-mcp-analytics branch July 30, 2026 17:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Capture MCP tool calls in PostHog - #128

Capture MCP tool calls in PostHog#128
masnwilliams merged 9 commits into
mainfrom
hypeship/posthog-mcp-analytics

masnwilliams commented Jul 29, 2026 •

edited by cursor Bot

Loading

Uh oh!

vercel Bot commented Jul 29, 2026 •

edited

Loading

Uh oh!

socket-security Bot commented Jul 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

dcruzeneil2 left a comment

Uh oh!

masnwilliams commented Jul 30, 2026

Uh oh!

dcruzeneil2 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

masnwilliams commented Jul 29, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Sessions

No payloads leave the server

Deliberately left off

Verification

Note

Uh oh!

vercel Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

socket-security Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dcruzeneil2 left a comment

Choose a reason for hiding this comment

Blockers

Application-wide exception capture is enabled

Failed tool responses can still leak through error properties

The payload filter permits unknown properties by default

Non-blockers

Analytics identity and grouping

The selected distinct_id may fragment user profiles

API-key traffic lacks a stable customer identity

Organization grouping is not included

Session and transport correctness

Legacy SSE sessions may be split

Protocol version is not preserved

Request reconstruction has no fallback

Performance

Large payloads are processed before they are removed

Recommendation

Uh oh!

masnwilliams commented Jul 30, 2026

Blockers

Non-blockers

Uh oh!

dcruzeneil2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

masnwilliams commented Jul 29, 2026 •

edited by cursor Bot

Loading

vercel Bot commented Jul 29, 2026 •

edited

Loading

socket-security Bot commented Jul 29, 2026 •

edited

Loading

The selected `distinct_id` may fragment user profiles